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A laboratory taught by a graduate assistant, a programed instruction textbook, 
and cofY.puter-assisted instruction (CAD were evaluated comparatively as 
supplemental instruction in a graduate course entitled "Statistical Methods in 
Education" at the State University of New York at Buffalo. The experimental 
population of 64 students was divided unevenly and at random to represent each of 
the three treatments. Evaluation was based on two factors related to course 
objectives: cognitive material learned and student "affect" determined by 

questionnaire. While the study revealed no significant differences among the 
supplementary media used, students received the programed materials more 
favorably than the laboratory. To strengthen experimental controls, a revised CAI 
program and additional tutorial help are indicated. (TD 
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AN EVALUATION OF THREE INSTRUCTIONAL MEDIA AS SUPPLEMENTS TO 
A GRADUATE COURSE IN EDUCATIONAL STATISTICS 

Traditionally, the introductory graduate statistics course offered 
by the Faculty of Educational Studies at the State University of New York 
at Buffalo (SUNYAB) supplements lectures with laboratory sessions conducted 
by a graduate assistant. It is believed that the laboratory sessions are 
desirable because they offer students certain learning opportunities that 
lectures do not permit; that is, opportunities to review topics covered in 
lectures and to ask questions related to homework problems. The intrinsic 
benefits of programned instruction (self«pacing, imnediate feedback, organi- 
zation by concepts, and content- specialist authors) suggest that this medium 
could also function effectively as a supplement to lectures. It was the 
purpose of this study to plan and execute a comparative evaluation of three 
media used as supplemental instruction in an Introductory graduate course 
in educational statistics. These media were: laboratory taught by a grad- 

uate assistant, programmed instruction textbook, and computer-assisted 
instruction. The evaluation was based upon two factors related to the 
course objectives: amount of cognitive material learned, and the degree of 

positive and negative effect associated with the media. 



A. Procedure 

1. Population 

The population in this study was defined as those students who take 
Statistical Methods in Education: Descriptive" under the present require- 

ments of the various departments within the Faculty of Educational Studies, 

It was not possible to detarmine how differences in department 
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requlrements within the Faculty affect the description of the population. 

No assumptions were made regarding the composition of this population with 
respect to variables affecting learning. That is, no data were available 
on intelligence quotient scores, previous statistics courses, and degree 
of mathematics sophistication. It was assumed, however, that the distribu- 
tions of these variables do not change substantially from year to year. 

2. Sample 

Under the foregoing assumptions, the class taught during the fall, 

1967, semester may be regarded as a random selection from the population 
under study. The original enrollment in the class consisted of seventy- 
five students, with twenty-five students randomly assigned to each treat- 
ment group. However, two students assigned to the computer-assisted 
Instruction (CAI) group were unable to commute to the University during 
the week to use the terminal and were reassigned to the programmed text- 
book group. An additional two students assigned to the CAT group chose 
not to use the medium and were dropped from the sample. One student as- 
signed to the laboratory also had commuting difficulties and was conse- 
quently reassigned to the programmed textbook group. By January, nine 
students had dropped the course. Thus, the total sample size for the 
analysis was reduced to sixty-four with distributions as follows: CAI 
group, nineteen students; laboratory group, twenty students; and programmed 
textbook group, twenty-five students. (For analysis of the final results, 
the total sample size and the distributions required further adjustment 
as discussed under S . Results . ) 

3. Criteria Measures 

The course objectives included familiarity with Introductory content 
in descriptive statistics and ability to manipulate the concepts presented. 
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Actual computation, while a specific goal, was not emphasized* 

Instruments (the final exam and the two mid-term exams) were created 
by selecting items from a pool of such items which had been developed 
from previously used and tested items. The test items had high discrimina- 
tion indices (greater than 1) and acceptable difficulties (between .40 and 
.60). 

To begin a long-range study of the attitudes of college students toward 
pxosi-amiued instruction, a series of open-ended questions was developed. To 
complement these questions, three yes-no questions were asked those stu- 
dents who received CAl. The questionnaire can be found in the Appendix. 

4. Treatments 

Three treatments were compared; these were a CAI program, a programmed 
textbook, and laboratory sessions conducted by a graduate assistant. 

The CAI program was an introductory course in descriptive statistics 
written by James Schoonard for Science Research Associates; it was avail- 
able through computer hook-up with an IBM 7010 central processor located 
in Yorktown Heights, New York. Console times were scheduled, and students 
were told that additional time would be available if they required it. 

After an orientation session, the students were told to follow the instruc- 
tions given in the CAI workbook. Proctors were on duty to handle scheduling 
problems and assist with console and communication problems. 

The programmed textbook was Basic Concepts in Statistics by McCollough. 
Students were instructed to complete the text at their own pace. 

The format of the laboratory conducted by a graduate assistant was 
not altered from that of previous semesters. It was informal and pre- 
sented so that questions and discussion related to either homework or 
lectures were welcomed. 
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Attendance at the lecture sessions was not compulsory. The lectures 

were given in conjunction with assigned readings in Weinberg and Schumaker's 

text Statistics: An Intuitive Approach . All students were required to 

complete six homework assignments. These assignments were corrected and 

* 

graded. If a student made major errors in these assignments, he was re- 
quired to correct them and hand in the corrected work. 

5. Design 

Analysis was conducted on both the cognitive and affective levels. 
Procedures for the analysis of each area were as follows: 

a. Cognitive 

The cognitive analysis was concerned with testing the hypothesis: 
Ho: There are no differences among treatment levels 

with respect to final grades. 

Accordingly, a one-way, fixed analysis of variance was run. The three 
levels of the treatment variable were CAI, programmed textbook, and grad- 
uate assistant-taught laboratory; the dependent variable was final grade. 
The final grade was a numerical composite score of the two mid-term exam 
grades and the final exam grade. The statistical tests were made at the 
.05 confidence level. Appropriate Scheffe post-hoc comparisons were to 
be made if the three group means proved significantly different. 

b. Affective 

The questionnaire given to the students is considered an initia- 
tive step in a long-range study of students' attitudes toward CAI. For 
this reason the main questions were open ended. 

The questionnaire was administered to the students after they had 
completed their final examinations. It was decided that students would 
be more likely to express their true attitudes if they were not required 
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to sign their names; therefore, the questionnaire was anonymous and stu- 
dents were encouraged to give spontaneous comments. Analysis of the 
questionnaire was a descriptive report of frequencies. 



B. Results 

1. Cognitive 

The analysis of variance produced an F of .03. The tabled F for two 
and sixty degrees of freedom at * .05 Is 3.93, Therefore, the hypothesis 
that there are no differences among levels with respect to final grades 
is not rejected. 

The results of the analysis of variance are given in Table 1. 



Table 1 



Analysis of Variance 



- 

Source 


d.f. 


M.S. 


F 


Between 


2 


27.81 


il 

1 1 

1 ^ 

1 O 

• 

1 

1 


Within 


60 


807.87 




Total 


62 


835.68 





Note.— One student assigned to CAI never reported and 
thus his grade was dropped from the sample. 



If it is desirable to make inferences to the population, it could be 
concluded that for the population, instructor, graduate assistant, and 
programmed materials studied, there are no significant differences between 
graduate assistant-taught laboratory, programmed textbook, and CAI as 
supplemental teaching media. 
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2. Affective 

The questionnaire requested students to check the group to vhlch they 
had been assigned. The frequencies shown in Table 2 indicated discrep- 
ancies between the assigned cell totals (after attrition) and the re- 
ported assignments. 

Table 2 



Discrepancies in Assigned Group and Student-Reported Group 





Assigned 


S tudent-Reported 


CAI 


19 


15 


Programmed Text 


25 


27 


Laboratory 


20 


20 


Total 


64 


62 



Since the questionnaires were anonymous, it is not known to what ex- 
tent the student reports correspond to original assignments. The following 
discussion, therefore, is based on the assumption that the medium checked 
is the medium at which the student focused his responses. The specificity 
of the responses in all cases indicates that this assumption is valid. 

The following discusses student responses for each of the media: 
a. Laboratory 

Of the twenty students who indicated that they had been assigned 
to uhe graduate assistant-taught laboratory, five students stated that 
they had attended the laboratory and fifteen students stated that they had 
never attended the laboratory. In response to the question asking for 




average amount of time spent per week In the laboratory, the five students 
who indicated they had used the laboratory reported the amounts of time as 
given in Table 3. 

Table 3 



Student -Reported Use of Graduate Assistant-Taught Laboratory 



Student 


Reported Time Used Laboratory 


Student 1 


Once for 30 Minutes 


Student 2 


Twice for 30 Minutes Each Time 


Student 3 


Weekly: 2 hr s. /week 


Student 4 


Weekly: 3 hrs./week 


Student 5 


Weekly: 3-5 hrs./week 



Reasons for lack of attendance at the laboratory are not known; how- 
ever, several unsolicited comments indicated scheduling difficulties. 

In response to whether they liked the laboratory and if they found the 
method effective, the five students commented as indicated in Table 4. 

Student 2 (see Table 3) reported that he also made use of the pro- 
grammed textbook, and another student who did not attend the laboratory 
indicated that he used the programmed textbook from two to four hours 
per week. 

b. Programmed Textbook 

Table 5 indicates the amount of time per week that students re- 
ported using the programmed textbook. It should be noted that the total 
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Table 4 

Frequencies of Students Reporting Effectiveness of 
Graduate Assistant-Taught Laboratory and Breakdown 
with Respect to Liking the Medium 



Found L*b S^^ective 


Did Not Find Lab Effective 


Liked 


Did Not Like 


Liked 


Did Not Like 


1 


1 




1 



Note.— One student did not find lab effective yet had ambivalent feel- 
ings with respect to liking and not liking the medium. 

One student indicated he found the medium both effective and 
noneffective. 



Table 5 



Reported Amount of Time Spent 
Using Programmed Textbook 



Number of Students 


TimeA7eek 


3 


0 hrs. 


7 


Less than 1 hr. 


7 


1 hr. 


6 


2 hrs. 


1 


3 hrs. 
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number of students is only twenty-four; two students did not respond to 
the question, and one student indicated that he used the programmed text- 
book but was unable to arrive at a time estimate* 

Student responses to the questions asking opinions as to effective- 
ness and liking of the programned textbook are summarized in Table 6* 

Table 6 

Frequencies of Students Reporting Effectiveness of 
Programmed Textbook and Breakdown with Respect 
to Liking the Medium 



Found P*T* Effective 


Did Not Find P*T* Effective 


Liked 


Did Rot Like 


Liked 


Did Not Like 


14 


2 


5 


6 


16 


] 


LI 



Table 7 Indicates subjects* comments on the programmed textbook* 

Among th( are responses to the questionnaire and unsolicited comments* 
They have been classified by content and positive and negative effect and 
grouped together under the broadest response. Thus, "easy to read and 
understand" includes "reduced the material to a basic level" and "simple." 
c • Computer-Assisted Instruction 

The fifteen students who reported being assigned to the CAI 
group all indicated that they had used the program* One student stated 
that he also received outside tutoring* The actual amount of student time 
spent working at the computer, as calculated from the print-outs, is shown 
in Table 8. It is difficult to tally average amount of time spent per 
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Table 7 



Student Comments Regarding Programmed Textbook 



Comments 


Frequency 


Positive Comments 




Easy to read and understand 


8 


Clarified concepts 


5 


Liked proceeding at own rate 


2 


Required little additional work 


1 


Convenient 


1 


Liked knowing answers immediately 


1 


Total 


18 


Negative Comments 




Steps too small 


5 


Not enough time 


4 


Confused by differing approaches 


3 


Net sure of effectiveness 


2 


Preferred scheduled time 


2 


Program confusing 


1 


Too fast 


1 


Total 


18 



u 




Note*— ”X*' means that the student completed the corresponding chapter 
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week working with the CAI console end with the workbook because some stu- 
dents Indicated the time varied and others reported the time decreased as 
they lost Interest In the medium* The range In time reported was from one 
hour per week for four weeks to ten hours per week for the semester* 

Responses as to the effectiveness and liking of CAI were mixed. It 
appears that students who thought CAI effective also liked it; students 
who believed CAI ineffective disliked the medium. The responses are shown 
In Table 9* 

Table 9 



Frequencies of Students Reporting Effectiveness of CAI 
and Breakdown with Respect to Liking the Medium 



Found CAI Effective 


Did Mot Find CAI Effective 


Liked 


Did Mot Like 


Liked 


Did Mot Like 


7 


1 


0 


7 


Total: 8 


Total: i 





Only three of the fifteen CAI students indicated difficulty In learning 
to type their responses. These three students also indicated that they 
found confusing the differences between the symbolic notation used in the 
CAI program and that used by the instructor. Of the twelve students who 
had no typing problems, eight found the symbolic notation confusing. Five 
students would have preferred a program having no workbook but requiring 
instead direct answering of all questions at the console. These responses 
are summarized in Table 10. 

Spontaneous positive comments of the CAI group to aspects of computer- 
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Table 10 

Response Frequencies to Questions Concerning CAI 



Question 


Response Frequencies 


Yes 


No 


Total 


Difficulties in Typing 


3 


12 


15 


Symbolic Notation Confusing 


11 


4 


15 


Prefer Full-Time Computer 


5 


10 


15 



assisted instruction were primarily concerned with the nature of the ma- 
terial programmed; however, the negative comments seem to indicate that the 
students expected more from the computer with respect to flexibility, 

amount in r<torage, and ability to respond. Students' coonents on CAI are 
given in Table 11. 

Examination of the computer print-outs shows that the eighteen stu- 
dents who were assigned to the CAI group and who completed all of the 
lecturer's course requirements spent, for the semester, an average of 
seven hours and nineteen minutes on the CAI console. Examination of Table 
B reveals that there was a great diversity in total amount of CAI time, 
ranging from over two hours at the console for student 17 to over eighteen 
hours for student 8. While there does not seem to be any relationship 
between total time spent at the console and final course grade, it should 
be noted that each of the four students receiving an A as a final grade 
spent at least five and one-half hours at the console. Table 8 also 
Indicates the number of chapters each student completed. Over half (ten) 
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Table II 



Student Conments Regarding CAl Course in Statistics 



Conments 


F rc^jusficy 


Positive Comments 




Indicates errors 


2 


Appreciated additional work 


2 


Forced to understand terminology 


1 


Helped with thinking in statistics 


1 


Appreciated guided lessons 


1 


Forced to answer questions otherwise neglected 


1 


Total 


8 


i Negative Comments 




Angry or frustrated with machine 


6 


j Would like to ask (machine) own questions 


4 


Machine not available 


4 


1 

j Machine explanations too lengthy 


1 


1 

1 Too time consuming 


1 


j Desires more time 


1 


Rejects correct answer 


1 


No flexibility In schedule 


1 


Sequence out of phase with class 


1 


Questions not applicable to class 


1 


Assumed too much sophistication in statistics 


1 


Total 


22 
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of the students completed the first seven CAI chapters* These chapters 
constituted the primary emphasis of the course as taught by the lecturer. 
Again there is no correspondence between chapters completed and grades; 

yet, one notices that each of the "A" students completed at least the first 
six chapters. 

Table 8 reveals what was probably one of the most important contribu- 
tions of CAI as a teaching medium; that is, it allows the student the op- 
portunity to go beyond what is offered in the classroom. Analysis of 
variance (ANOVA) was the topic of the last class lecture. Students re- 
ceived neither homework nor examination questions pertaining to this topic. 
There was no course- oriented reason for any student to pursue the topic. 
Yet, one of the CAI students completed the ANCJVA chapter in the CAI. 
Although he could have borrowed an ANOVA book from the library, it may be 
that the CAI program allowed him to continue his study in a program that 
was geared to his ability and that used familiar notation. 

C. Discussion 

While this study did not reveal significant differences among the 
supplementary media used, student comments on the questionnaire indicated 
that the programmed materials were more favorably received by the students 
than was the laboratory. Many students seemed to appreciate the tutorial 
method provided by the programmed textbook and CAI. Much of the criticism 
of CAI was not directed at the concept of computer-assisted Instruction, 
but rather at the limitations of the program and scheduling difficulties. 
Because this statistics CAI program was a prototype, these program criti- 
cisms are to be expected; also the Instructional Communication Center has 
gained valuable information as to appropriate mechanical and clerical 
procedures for running a CAI course. 



It is pertinent to this discussion to mention several factors which 
may have affected the experimental results. Many unsolicited comments on 
the questionnaire praised the instructor for his fine teaching. Several 
students commented that the statistics course had been one of the best 
courses they had taken at the University. It is very possible that the 
instructor's Influence in this situation constituted the actual source of 
treatment effect. Further, it cannot be assumed that within each level 
of treatment all students received the same amount of treatment. It was 
possible to eliminate the one student who did not use CAI$ however. Table 
8 clearly shows that many students did not complete the first eleven chap- 
ters. Table 5 shows the wide variety of time devoted to the programmed 
textbook. The limited use of the graduate assistant-taught laboratory 
suggests that this group functioned primarily as a control group. In 
addition to the varying amounts of treatment, three students admitted to 
having used an additional supplementary medium than that to which they 
had been assigned. Also, a graduate student in the educational statistics 
and evaluation area reported that she tutored three students in this intro- 
ductory course. Only one student reported receiving tutorial help; it is, 
therefore, not known how many students in this sample actually did seek 
tutorial assistance. 

In conclusion, the results of this study suggest replicating the 
study using stricter controls, such as another instructor and graduate 
assistant and another or a revised CAI program. 



APPENDIX 



F 521 Questionnaire 



The following questions are related to the supplementary Instructional 
media available to you this semester. Your responses to this question^ 
naire will constitute one phase of evaluative research; therefore, please 
seriously consider your answers before responding to Insure the integrity 
of the results. Do NOT sign your name. 

1. Check the teaching group you were assigned to: 

computer-assisted instruction 

programmed textbook 

graduate assistant-taught laboratory 

2. If you made use of more than one of these Instructional media, please 
list them below. 



3. Indicate the average amount of time per week you devoted to the in- 
structional medium to which you were assigned. 

4. Did you find the instructional medium to which you were assigned 
effective as a learning method? Please explain. 

Yes No 



5. 



Did you like the type of instructional medium to which you were 
as&igned? Please explain. 

No 



Yes 



Answer the following questions only if you were assigned to the CAI group. 

6. Did you have trouble learning to type in your responses? £j Yes /"7 No 

7 . Did you find the differences between the symbolic notation used in the 

CAI program and in class confusing? Yes /~7 No 

8. Would you have preferred a program that did not have a workbook, but 
instead required you to directly answer all questions at the console? 

n Yes n No 



